The invention relates to increasing the quality of high-resolution images and more particularly to increasing resolution at fractional pixel positions for a particular scene (called the reference image) by using multiple images of the scene.
Increasing the pixel resolution of an image beyond the resolution of the imaging sensor via digital postprocessing using multiple images provides valuable means of obtaining high quality images with cameras equipped with inexpensive low resolution sensors, or exceeding the physical capability of any given sensor and obtaining higher quality images.
Different single-image interpolation techniques are used to increase the amount of pixel information used to represent an image. Linear interpolation techniques do not increase the actual information content of an image but simply increase the number of pixels and lines in the image. Nonlinear interpolation techniques utilize a priori information about the image structure (e.g., direction of edges, and image object geometry) and in some instances, may provide better results than linear interpolation.
Referring to FIGS. 1 and 2, multiple images 16 of a scene are used to improve image resolution. The multiple images 16 may be individual shots acquired by a digital still camera, or successive frames/fields acquired by a video camera. New image information is contained in image samples 17 of the different images 16 that are inter-related by relative motion. This method is described in A. M. Tekalp, M. K. Ozkan and M. I. Sezan, "High resolution Image Reconstruction from Lower-resolution image sequences and space-varying Image Restoration", IEEE International Conference on Acoustics, Speech and Signal Processing, San Francisco, Calif., Vol. III, March 1992, pages 169-172.
In this method, a reference image 12 is first chosen from the multiple images 16. Motion information includes a motion vector field 14 estimated from a low resolution image 16 onto the reference low resolution image 12. Each motion vector field 14 represents the relative displacement from image 16 onto the reference image 12. Image samples from image 16 are mapped onto the reference image 12 to create a high-resolution image 19 using the motion vectors 14. Image 19 is a high resolution version of the scene captured in the reference image 12. New image samples derived from the other low-resolution images 16 are shown by "x" in the high resolution image 19.
The low-resolution reference image 12 may not be able to capture image detail faithfully, such as image detail 10 in the neighborhood of the low-resolution pixel samples 17 in the reference image 12. This inability to represent detail is a direct consequence of the Nyquist Theorem for one and multi-dimensional sampled signals which states that any detail being at a frequency equal or higher than half the sampling rate cannot be faithfully represented in image 12. However, due to camera motion while electronically capturing the images 16 or motion in the image taken by the camera at different times, image detail 10 might be re-constructed unambiguously through the additional image information revealed in one or several of the low-resolution images 16. The high-resolution image 19 uses the low-resolution samples 17 from the other images 16 to re-construct the additional image details 10.
Referring to FIG. 2, intersection of dashed lines 18 indicate locations of the additional sampling grid points 20 (pixels) that are used to increase the resolution in reference image 12 beyond its current resolution level identified by squares 17. As depicted in FIG. 2, the samples x from the other low resolution images 16 are mapped, in general, to arbitrary inter-pixel locations that do not coincide with any high-resolution inter-pixel location 20. Sample locations 20 constitute a uniform high resolution sampling grid. Producing new samples at these locations is the ultimate goal of any resolution improvement technique since all image display devices operate on the basis of a uniform sampling grid. The original low-resolution samples 17 and the new samples x constitute samples of the higher resolution image over a non-uniform sampling grid.
A very complex interpolation process is required to derive pixel values for the high-resolution image 19 at uniform grid locations 20 from the non-uniformly located samples x. For example, multiple samples 21 must be concurrently used by a multi-dimensional digital filter to generate the pixel value at the high-resolution grid point 20A. Typically, samples at grid locations 20 cannot capture maximum image details due to limitations in the size of the digital filters used for interpolating the samples x to the location 20A. In addition, there is also no guarantee that there be any samples x in the region of support of the digital interpolation filter and as a result, no further image quality can be produced when this occurs.
Accordingly, a need remains for producing high-resolution images by using samples taken from other images while increasing the quality of the high-resolution image and reducing the complexity of the process used to generate the high-resolution image.